The present invention generally relates to a user interface in an interactive application environment and, more particularly, to an agent character supported user interface.
The present invention especially relates to an agent layout determination method in a xe2x80x9cthree-dimensional space (a virtual reality space and a mixed reality space obtained by superposing the virtual reality space onto a real space)xe2x80x9d, which does not impose negative influences (coerced and anxious feelings) on the user.
Along with the advance of multimedia information presentation technologies, huge volume of information is presented to the user and it may often become difficult for the user to understand what kind of request is needed to be input to a computer or what is required of him or her by the computer system.
Under these circumstances, conventionally, Windows as an operating system available from Microsoft Corp., or Word as a wordprocessing application program displays an agent character to give a guidance to the user in response to a user request or when a predetermined condition for the program is satisfied.
An agent character provided by such program is the one for a display environment of a two-dimensional image, and does not consider the gazing position of the user on the display screen.
As known technologies pertaining to agent characters, for example, Japanese Patent Laid-Open No. 9-153145 (Laid-Open date: Jun. 10, 1997), xe2x80x9cComputer which sees, hears, and talks with smile (Multimodal interactive system)xe2x80x9d, Electrotechnical Laboratory Press Meeting Reference, and Lewis Johnson, Jeff Rickel, Randy Stiles, and Allen Munro, xe2x80x9cIntegrating Pedagogical Agents into Virtual Environmentsxe2x80x9d, Presence, Vol. 7, No. 6, pp. 523-546, are known.
An xe2x80x9cagent display apparatusxe2x80x9d disclosed by Japanese Patent Laid-Open No. 9-153145 is a technique for agent display implemented as a user interface which makes processes in correspondence with user""s purpose, favor, and skill level. More specifically, a plurality of types of agent states (appearances (costumes of agent), modes of expression, and emotional expressions) are prepared, and a combination of agent states is determined in correspondence with user""s purpose, favor, and skill level, thus displaying an agent character in the determined combination.
However, the technique disclosed by this reference does not essentially consider any three-dimensional space.
xe2x80x9cComputer which sees, hears, and talks with smile (Multimodal interactive system)xe2x80x9d, Electrotechnical Laboratory Press Meeting Reference was released at a home page having the world wide web URL address xe2x80x9cwww.etl.go.jp/etl/mmpress/mmpress.htmlxe2x80x9d (updated Nov. 9, 1996), and pertains to an agent implementation technique present in a three-dimensional space. More specifically, this paper is a study about a prototype agent which comprises real-time image recognition, speech recognition, and speech synthesis functions. This paper aims at exploring specific specifications that a computer must satisfy to implement smooth interactions, and acknowledging technical problems to be solved. In this paper, an agent is present in a virtual space in a display, and interacts with the user via the window (screen). However, in this reference, the agent is always displayed at a given position, and never changes its position in consideration of its position relative to the user.
As for an agent character display technique in a three-dimensional space as an objective of the present invention, xe2x80x9cIntegrating Pedagogical Agents into Virtual Environmentsxe2x80x9d discloses a study of a system which reconstructs a life-size virtual three-dimensional work space, and allows the user to learn a task process with the help of an agent. When the user who wears an HMD selects a given menu item, the agent responds by synthetic voice. When the user selects a task demonstrate menu (Show Me!), the agent responds by its actions.
However, in this prior art, the agent makes only predetermined actions, and the display layout of the agent character does not consider relative position between the user position and the display position of the agent character.
In order to realize a user interface in a display environment of a three-dimensional image, the agent character must also be displayed as a three-dimensional image. That is, the agent character is also given a three-dimensional display position and scale.
FIG. 1 explains a case wherein the agent character given a certain scale value is displayed in an unexpected size and surprises the user. More specifically, since the agent character is given a three-dimensional display position and scale in a three-dimensional image space, if the user wants to communicate with the agent character at a close distance in a three-dimensional display environment in which the agent character is given the same size as that of a human being, a program that presents the agent character displays the xe2x80x9cfacexe2x80x9d of the agent character in the same size as that of a human face, as shown in FIG. 1. In some cases, the face of the agent character is displayed in a full size of the display screen, and makes the user feel anxious or coerced.
Or even when the display size of the agent character is not so large, the agent character may occlude an object the user wants to see depending on the viewpoint position of the user, as shown in FIG. 2.
Such problems do not normally occur in a two-dimensional display space, and occur only in a three-dimensional display space when the viewpoint position/posture of the user, and the position of the agent character are not taken into consideration.
The aforementioned Electrotechnical Laboratory Press Meeting Reference does not pose such problems due to limitations, i.e., since the agent is always displayed at a predetermined position.
Also, in xe2x80x9cIntegrating Pedagogical Agents into Virtual Environmentsxe2x80x9d, since the agent makes only predetermined actions, as described above, the above problems do not occur.
In other words, in an environment in which a three-dimensional agent character freely moves in a three-dimensional space, the aforementioned prior arts cannot avoid the problems shown in FIGS. 1 and 2.
The present invention has been proposed to solve the conventional problems, and has as its object to provide a user interface or an agent character layout determination method, which considers position relative to the user so as not to impose negative influences on the user.
In order to achieve the above object, according to the present invention, a user interface method which three-dimensionally displays a character designated by a program, and allows the user to interact with a computer system while communicating with the displayed character, comprises the steps of:
estimating a region of interest of the user;
estimating a display purpose of information in an application program; and
determining a three-dimensional display pattern of the character by applying a rule, which aims at preventing display of the character from being against user""s interest, to the estimated region of interest of the user and the estimated display purpose of information.
By considering the region of interest of the user, e.g., the position or line-of-sight direction of the user, the user can be free from any negative influences.
According to a preferred aspect of the present invention, in the rule a reference value used to determine if display of the character is against user""s interests is changed in accordance with a distance from the user.
According to a preferred aspect of the present invention, in the rule a reference value used to determine if display of the character is against user""s interests is changed in accordance with a distance from the user on a front side of a line-of-sight direction of the user.
According to a preferred aspect of the present invention, display of the character is inhibited in a first zone which has a distance from the user less than a first distance, and display of the character is permitted in a second zone which has the distance from the user that falls within a range from the first distance to a second distance larger than the first distance, and a third zone which is farther than the second distance.
According to a preferred aspect of the present invention, in the rule when the user and character do not communicate with each other, the character stays in the third zone which is farther than the second distance.
According to a preferred aspect of the present invention, in the rule when the user and character communicate with each other, the character is located in the second zone.
According to a preferred aspect of the present invention, in the rule the position/posture of the user are detected, and the character is located so as to prevent the region of interest from being occluded by the character from the detected position/posture of the user.
According to a preferred aspect of the present invention, in the rule the position/posture of the user are detected, and the character is located so as to prevent the character from being occluded by the region of interest from the detected position/posture of the user.
According to a preferred aspect of the present invention, in the rule the character is located not to fall in the region of interest.
According to a preferred aspect of the present invention, in the rule when an object to be manipulated by the character is present in the region of interest, the character is located within a range in which the object is accessible.
According to a preferred aspect of the present invention, if there are a plurality of different layout positions of the character, which satisfy the rule, the character is located at a position which can minimize a distance between the region of interest and the layout position of the character.
According to a preferred aspect of the present invention, if there are a plurality of different layout positions of the character, which satisfy the rule, and the region of interest is not present, the character is located at a layout position closest to the user.
In some cases, a plurality of layout candidates may be present depending on the two layout conditions.
According to a preferred aspect of the present invention, if there are still another plurality of different layout positions of the character, which satisfy the rule, the character is located at one of the plurality of different layout positions.
According to a preferred aspect of the present invention, if a layout position that satisfies the rule is not present, the character is located outside a field of view of the user.
According to a preferred aspect of the present invention, the rule further considers a size of the character.
According to a preferred aspect of the present invention, the rule further considers an attribute of the character.
According to a preferred aspect of the present invention, the size is determined in advance in correspondence with the character.
According to a preferred aspect of the present invention, the attribute is determined in advance in correspondence with the character.
According to a preferred aspect of the present invention, the rule further considers a situation of the character.
According to a preferred aspect of the present invention, the situation of the character considers if the character is waiting for an instruction from the user, having conversation with the user, or is automonously moving.
The above object can also be achieved by a storage medium which stores a program code for implementing the user interface or an information processing apparatus which stores and executes that program code.
Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.